Image Pre-processing with a Generalised Grey Scale N Tuple Operator

This paper describes a generalisation of the binary N tuple technique originally described by Bledsoe and Browning (1). The binary N tuple technique has commonly been used for the classification (2) and pre-processing (3) of binary images. The extension to the method described here allows grey level images of objects to be classified using the N tuple method without first having to convert the image to an intermediate binary representation. The paper illustrates the methods use in image preprocessing. hhhhhhhhhhhhhhhhhh From : Lecture Notes in Computer Science (301). Springer Verlag. Ed. J Kittler. Pattern Recognition, 4th international Conference, Cambridge, UK, March 1988. Introduction The binary N tuple pattern recognition process was originally described by Bledsoe and Browning (1), and has been applied to a number of image processing tasks such as character recognition (4), face recognition (2), and scene analysis (5). The N tuple process may be seen as a simple perceptron (8) with a non linear pre-processing transform. Thus it is an adaptive classifier, which must be trained on a subset of the patterns to be recognised. It has two major advantages over the perceptron, first, it learns each training pattern quickly and secondly it is able to classify patterns that fall into the category of ‘exclusive or’ problems (9). However the N tuple method has always been limited to processing binary images; grey scale images may only be processed by the method if they are first converted to a binary representation (see 3). The method described here requires no intermediary binary representation of the grey scale data. Although the binary N tuple process has been shown to be adequate in a number of applications there are a number of cases where it is insufficient, requiring the full grey scale information to be used in the classification process. For instance in the classification of edge features where it is necessary to determine the angle and ‘sharpness’ or slope of an edge. This information is only present in the grey scale domain. Binary N tuple process The binary N tuple process as described by Bledsoe and Browning (1) and Aleksander (2) may be seen as a two stage process. The pre-processing stage performs a non-linear transform on the input image. The resultant image is then processed using a simple perceptron with binary weights. The addition of the preprocessor allows the perceptron (a linear classifier) to classify non-linearly separable data (i.e the ‘exclusive or’ or parity problem). The binary N tuple process takes as its input a binary image. From this image a set of tuples are formed. Each tuple is made up of N elements from the image. The origin of each pixel in the image used to make up each tuple is defined once, normally the origin of each pixel is selected in a random manner, each pixel only contributing to one tuple. The optimal size of N depends on the characteristics of the data and the generalisation properties required (1). Each tuple may be denoted Dk,N . For each image there are N max number of tuples where N max = i/N. i being the number of elements in the image. scale=90 # dots per inch box invis wid 580 ht 235 at 290,118 # LayoutRectangle box invis wid 5 ht 5 at 0,0 # grid "α" at 461,167 line from 435,170 to 430,165 line from 430,175 to 435,170 "Fig 1: One N tuple processing unit, U." at 316,32 "1" at 89,157 "0" at 89,177 "1" at 89,197 "B()" at 182,167 line from 390,170 to 435,170 line from 100,160 to 160,160 line from 100,180 to 160,180 line from 100,200 to 160,200 line from 80,150 to 100,150 line from 80,170 to 100,170 line from 100,150 to 100,170 line from 80,150 to 100,150 line from 80,170 to 80,150 line from 100,190 to 100,170 line from 80,190 to 80,170 line from 80,210 to 100,210 line from 100,190 to 100,210 line from 80,190 to 100,190 line from 80,210 to 80,190 line from 200,160 to 240,160 line from 200,180 to 240,180 line from 200,200 to 240,200 line from 280,160 to 355,170 line from 280,180 to 355,170 line from 280,200 to 355,170 line from 365,165 to 380,160 line from 370,170 to 365,165 line from 365,175 to 370,170 line from 380,180 to 365,175 box wid 35 ht 30 at 372,170 line from 245,165 to 270,170 box wid 10 ht 10 at 275,160 box wid 10 ht 10 at 245,160 line from 245,185 to 270,190 box wid 10 ht 10 at 275,180 box wid 10 ht 10 at 245,180 line from 245,205 to 270,210 box wid 10 ht 10 at 275,200 box wid 10 ht 10 at 245,200 line from 200,95 to 240,95 line from 245,100 to 270,105 box wid 10 ht 10 at 275,95 box wid 10 ht 10 at 245,95 box wid 40 ht 125 at 180,148 line from 280,95 to 355,170 ellipse wid 5 ht 5 at 257,143 ellipse wid 5 ht 5 at 257,123 "Tuple" at 45,197 "D" at 74,132 "k,N" at 97,122 "1" at 229,207 "2" at 229,187 "3" at 229,167 "8" at 229,102 "Switches" at 276,222 scale=90 # dots per inch box invis wid 565 ht 365 at 282,183 # LayoutRectangle box invis wid 5 ht 5 at 0,0 # grid "Processors" at 185,337 "Tuple" at 80,337 "n" at 279,112 "2" at 254,212 "1" at 274,262 "α" at 256,117 "α" at 231,217 "α" at 256,272 "Fig 2: An N tuple classifier." at 286,32 "R" at 389,202 "n" at 59,117 "2" at 59,197 "1" at 54,287 "," at 49,117 "," at 49,197 "," at 49,287 "k" at 44,117 "k" at 44,197 "k" at 44,287 "n" at 184,112 "2" at 184,197 "1" at 184,282 "U" at 174,117 "U" at 174,207 "U" at 174,292 "D" at 34,122 "D" at 34,202 "D" at 34,292 line from 375,205 to 370,200 line from 370,210 to 375,205 line from 340,205 to 375,205 line from 305,195 to 325,190 line from 315,205 to 305,195 line from 305,215 to 315,205 line from 325,220 to 305,215 line from 290,205 to 215,115 line from 215,205 to 290,205 line from 290,205 to 215,290 line from 95,115 to 150,115 line from 95,205 to 150,205 line from 95,290 to 150,290 box wid 25 ht 60 at 82,115 box wid 25 ht 60 at 82,205 box wid 65 ht 60 at 182,115 box wid 65 ht 60 at 182,205 box wid 50 ht 50 at 315,205 box wid 65 ht 60 at 182,290 box wid 25 ht 60 at 82,290 Fig. 1 shows one tuple and the processing associated with it (N = 3). The first stage B( ) assigns one state, S, to the tuple in accordance with table 1*. The tuple size, N, equals 3 in this case, giving a total of 2 possible states for each tuple. iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii State Table iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii State Tuple element values iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii i j j iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 1 0 0 0 iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 2 0 0 1 iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 3 0 1 0 iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 4 0 1 1 iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 5 1 0 0 iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 6 1 0 1 iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 7 1 1 0 iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 8 1 1 1 iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii cc c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c cc c c c c c c c c c c c c c c Table 1 : The state table for the binary N tuple function B(), for an N tuple size of 3. The next stage in fig. 1 shows a set of switches. Each switch is used to record the occurrence of a particular state, S, assigned by B( ). There are 2 switches to record all the possible states of the tuple. The output from the switches are OR’ed by the summing unit to produce an output α. For each of the N max tuples there is one of the processing units, U, shown in fig 1. Fig. 2 shows how the outputs of all the units are summed to produce an output R, the response for the classifier. There are two phases of operation of the system, teaching and testing. Teaching causes the system to learn an input pattern so that during subsequent presentation and testing it may be recognised. When a pattern is presented for teaching, the image is broken into N tuples and each tuple assigned a state, S,as described above. Each state is recorded on the switches shown in fig. 1 by closing the switch connected to the line associated with the tuple state. As further patterns are taught into the system more switches are closed. Once a number of patterns have been taught, classification on an unknown pattern may take place. This process, called testing, uses the summing devices shown in fig. 1 and 2. First the image is broken into tuples and the function B( ) applied as during teaching. A state, S, will now be assigned to each tuple. This is reflected in fig. 1 as an output line of the B( ) processor going to 1. If the active output wire coincides with a closed switch, a logical 1 is passed to the output of the processing unit, α n. This is repeated for all tuples and the total number of α at logical 1 summed. This value appears as a response, R, as shown in fig 2. This value represents how well the input matches the previous images presented. Normally a number of classifiers are used, one for each pattern class to be classified. The classifier which gives the largest R value indicates the class of the input pattern. The process of teaching and testing may be defined formally. P r,i : The r’th input pattern P of size i. P r,i = 1 or 0 in the binary N tuple process. Dk,N : Pattern P is broken into k vectors or tuples each of N in size, where k max = i/N and k min = 0. B( ) : Some function that assigns one state S to each N tuple group, where S min = 0 and S max = 2 N in the binary N tuple process. W k,t : A vector of weights, where t min = 0 and t max = S max. W k,t is 0 or 1. hhhhhhhhhhhhhhhhhh * The function B( ) may be defined formally. If the N tuple, Dk,N , is represented as a binary (base 2) number, the state of the tuple is given as its decimal (base 10) equivalent. Teaching : the image P is supplied and transformed into the tuples D. Then W k,t = 1 where t = B(D k,N) for all k, k min to k max. This is repeated for all k patterns P in the training set. Testing : again the image P is transformed into the tuples D. Then R = k = 0 Σ k = kmax W k,t where t = B(Dk,N ). (A) where R is the response from the classifier. One classifier as described above may be used to discriminate between two patterns. More are used to classify a large number of patterns. For more details of the N tuple process and its generalisation and classification properties see (1,4,9). The heart of the binary N tuple process is the function B( ) which translates a given N tuple group into a state S. The function is exactly described by the state table in table 1; the function is inherently binary. To be able to process grey scale patterns using the N tuple process it is necessary to replace the function B( ) used in the binary process for a function that will take grey scale values. Grey scale N tuple processing The grey scale N tuple process described here was originally described by Austin (5) where it was used as a pre-processing operation to an associative memory in a scene analysis system. It is described here as a general form of grey scale N tuple processing. In grey scale images each pixel may take on any integer value between 0 and I max, where I max > 1. If a set of tuples is formed from such an image as described above the tuple element values will be non binary and thus the binary N tuple process cannot be used. It is a simple matter to threshold the N tuple at some pre-defined or actively defined value, and use the binary N tuple function B( ) to assign one of 2 states to the N tuple. For example the grey scale N tuple 4,4,2,1 could be thresholded at 2 to generate the binary N tuple 1,1,1,0. generating the state 14 after application of B( ). However, if the threshold is set to 3 the binary N tuple would become 1,1,0,0 giving the state 12 after application of B( ). The sensitivity of this method to the level chosen for the threshold rules it out as a practical method. For instance, the method would not cope with variations in over all light levels between teaching on one example pattern and testing on another. Even if the image that was tested was exactly the same apart from the over all light level, the classification process would fail. To over come this it is possible to teach all example patterns over all light levels. However, this would be a slow and inaccurate method. To overcome this problem the following Grey level state assignment function G( ) has been developed which takes a grey level N tuple G ′ k,N and assigns it to one state S′. Consider a general 4 tuple which has the elements A,B,C,D that has been derived from a grey level image. If A,B,C and D can take on the values Imin to Imax, then the N tuple may be sorted into a list of descending element values, this ordering will represent one state, S ′, of the N tuple. For instance consider the N tuple 4,3,1,2. We assign the labels A,B,C,D to each element respectively, then sort the tuple by value to produce the list A,B,D,C. The order of this list represents the state of the N tuple. The state is equivalent to the state of the binary N tuple produced by the function B( ). Each tuple in the image is sorted in this way, each particular ordering of a tuple represents one state, S′. The total possible number of states an N tuple may be assigned is a function of the tuple size, N, and is given by S′ max = k = 1 Σ N k! S 2 (N,k) (B) where S 2 (N,k) is the Stirling number of the second kind and is given by S 2 (N,k) = I K L S 2 (N − 1,k − 1) + k.S 2 (N − 1,k) thereafter 1 if k = 1 or k = N An example of all the possible states of a 3 tuple is given in table 2. i iiiiiiiiiiiiiiii state tuple order i iiiiiiiiiiiiiiii 1 A/B/C i iiiiiiiiiiiiiiii 2 A/C/B i iiiiiiiiiiiiiiii 3 B/A/C i iiiiiiiiiiiiiiii 4 B/C/A i iiiiiiiiiiiiiiii 5 C/B/A i iiiiiiiiiiiiiiii 6 C/A/B i iiiiiiiiiiiiiiii 7 AB/C i iiiiiiiiiiiiiiii 8 C/AB i iiiiiiiiiiiiiiii 9 A/BC i iiiiiiiiiiiiiiii 10 BC/A i iiiiiiiiiiiiiiii 11 B/AC i iiiiiiiiiiiiiiii 12 AC/B i iiiiiiiiiiiiiiii 13 ABC i iiiiiiiiiiiiiiii c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c Table 2 The possible states of a grey scale tuple of size N = 3. Each element of the tuple is labelled A,B and and C respectively. The notation AB/C indicates that the values of A and B are equal, and greater than C. In the case ‘ABC’ all tuple elements are equal. To perform grey scale N tuple processing the function G( ) replaces the function B( ) in the formal definition of teaching and testing. Exactly the same procedures may then be used, only now a grey scale image is supplied. Reduction of the Number of States Although the grey scale N tuple function G( ) described above may be used in grey scale image classification it suffers from a number of problems. The first of these relates to the number of states each grey scale N tuple can generate by the application of G( ) (i.e S ′ max), and the subsiquent amount of storage needed in a classifiaction system using the method. As an example a typical 256 by 256 pixel image would be broken into 16384 tuples where the tuple size is 4. Each tuple generates a possible 75 states, thus one classifier would require 75 * 16384 switches, approximately 1.2 x 10. A typical implementation (2) of the N tuple method uses 1 bit of conventional computer memory to store the state of one switch, thus one classifier would require 150 k bytes of storage. Furthermore, it can be seen from equation B that the memory used raises factorially with the size of tuple. This storage requirement may be reduced using a number of data packing techniques (6). However there is a simple extension to the method that allows a reduction of the number of states each tuple may be assigned, which uses the concept of ‘ranks’. A rank here is used to define a threshold region between which some values may lie. i.e. if a rank had an upper value of 4 and a lower value of 2 the rank interval would be 2 to 4. There may be many (non overlapping) ranks between two values X and Y. The number of ranks is denoted ρ. At the extreme this extension reduces the function G( ) to be exactly equivalent to the function performed by the binary N tuple function B( ). The state assignment procedure for reducing the number of states a tuple may be assigned is best explained with the aid of an example. Consider the following grey level 4 tuple : A = 8, B = 7, C = 2, D = 1. The following procedure is followed to find the state the tuple belongs to; i) Find the maximum responding tuple element, Max. and the minimum responding tuple element, Min. i.e Max = A = 8, Min = D = 1. ii) Find the ‘rank interval’, I, I = ρ Max − Min hhhhhhhhhhh Where ρ is the number of ranks required i.e. 2 8 − 1 hhhhhh = 3.5 iii) From these values a ranking table can be constructed, which contains ρ ranking intervals or threshold regions of size I. The tuple elements are then placed within these regions depending on the elements values. i.e iiiiiiiiiiiiiiiiiiiiiiiiiiiiiii Ranking Table iiiiiiiiiiiiiiiiiiiiiiiiiiiiiii Ranking interval tuple elements iiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 4.5 to 8 A, B iiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 1 to 4.5 C,D iiiiiiiiiiiiiiiiiiiiiiiiiiiiiii c c c c c c